Thanks for the comments @jafermarq! It's still a WIP, so lot of stuff have to be adjusted :)

If you'd like to add some plots to your readme, please do so in a new directory named _static placed in the same directory as your README.

Got it!

We think it's better not to include the logs from the experiments (I see currently you are including the outputs/ directory.

Yeah, first time using hydra and I didn't realized it created output files. Now everything should be fine (I've disabled the creation of files from base.yaml).

Ideally, all the dependencies needed are added to the pyproject.toml so the environment can be built easily by simply doing poetry install. Depending on what packages you use, this might get tricky to achieve actually. If that's the case, it's fine provide additional instructions on how to install those packages after doing poetry install. Then you can install additional ones via the standard pip install <> method.

I've set up the environment in poetry now. I will check if everything works once I've finished to integrate the code by creating a new environment from scratch.

Ping me once there are some minimal instructions on how to setup the python environment and how to run an experiment. From that I can take a closer look at the code. Happy to discuss about your baseline anytime.

Got it! Thanks again.

Hi @jafermarq, the baseline should be ready. Let me know if there are any issue or suggestions, I'll take care of them.

Hi @jafermarq, everything is ready. However, a test is not passing. Can't get why

@jafermarq do you know what's the problem with the test?

@jafermarq do you know what's the problem with the test?

All seems good now. I'll review soon.

@jafermarq do you know what's the problem with the test?

All seems good now. I'll review soon.

Thanks! :)

@jafermarq Thanks for the review! Hope the results are pretty much the same as mine.

I've addressed your issues and realized that if the .sh script is run instead of launching batches of experiments individually, you'll get all results into one file. At the same time, the notebook expects that each experiment has a different one (i.e. all_results_[dataset]_[flanders/no_flanders].csv).
Now the notebook should automatically organize results into the files it expects.

Hope to hear from you soon

@jafermarq Thanks for the review! Hope the results are pretty much the same as mine.

I've addressed your issues and realized that if the .sh script is run instead of launching batches of experiments individually, you'll get all results into one file. At the same time, the notebook expects that each experiment has a different one (i.e. all_results_[dataset]_[flanders/no_flanders].csv). Now the notebook should automatically organize results into the files it expects.

Hope to hear from you soon

Thanks! but to generate the results you show in the read me should I run the two commands in the Expected Results section or should I run those in run.sh instead? I'm doing the former but please let me know.

Thanks! but to generate the results you show in the read me should I run the two commands in the Expected Results section or should I run those in run.sh instead? I'm doing the former but please let me know.

Good point, that's an oversight from my side.

The best way --now that I've updated the notebook to divide the results automatically-- is to run the bash script. For this reason, I removed the former way to run the experiments to avoid mistakes. However, if you've already launched the experiments with that method it's perfectly fine, note that you should run them also for MNIST though.

By the way, as you suggested, I updated the main.py to create the output dir if not present.

@jafermarq